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Abstract: This study investigates the use of screening assessments within the increasingly popular Response to Interven¬ 
tion (RTI)framework, specifically seeking to collect concurrent validity evidence on one potential new screening tool, the Inde¬ 
pendent Reading Level Assessment (IRLA)framework. Furthermore, this study builds on existing literature by disaggregating 
the validity evidence across grade, program, and race/ethnicity to better understand how the assessment functions amongst 
varying demographic categories. We add to the limited research base of evidence that the IRLA tool may be an important 
instrument for bridging the gap between screening and providing intensive, systematic instruction as detailed by the What 
Works Clearinghouse (Gersten et al, 2008). 


T he use of Response to Intervention (RTI) has be¬ 
come increasingly popular in schools since its rec¬ 
ommendation by the Individuals with Disabilities 
Education Act (IDEA) reauthorization in 2004. RTI is a 
multitier approach to support students with learning and 
behavior needs, emphasizing high-quality, scientifically 
based instruction, and ongoing student assessment (RTI 
Action Network, n.d.). In the last decade, the use of the 
framework has proliferated. In 2011, 94% of respondents to 
the RTI Adoption Survey reported their schools were at some 
stage of RTI implementation, while 68% of respondents 
were either in process or full implementation districtwide, 
up from only 24% in 2007 (Castillo & Batsche, 2012; 
Spectrum-K12, 2010; 2011). Elowever, less is known about 
how RTI screening tools can serve as an important in¬ 
strument for providing additional instructional supports. 

This study investigates the use of screening assessments 
within the RTI framework. More specifically, we seek to: 
(a) collect concurrent validity evidence on one potential new 
screening tool, the Independent Reading Level Assessment 
(IRLA; American Reading Company, n.d.) framework that 
identifies which students need placement in RTI to improve 
their academic and behavioral skills; and (b) to document 
one district’s journey, under the RTI framework, to find 
a screening tool that best matched the district’s unique 
needs. We extend the literature further by disaggregating 
the validity across grade, program, and race/ethnicity to 
better understand how the assessment functions amongst 
varying demographic categories. 

Background 

The use of universal literacy assessment (i.e., screening) 
has surged in conjunction with the rise in popularity of 
RTI (Fuchs, Fuchs, & Compton, 2012). “The primary pur¬ 
pose of screening in an RTI framework is to identify those 
students who without further intervention will be likely to 
develop reading problems at a later time” (Johnson, Pool, 
& Carter, 2011, p. 1). Screening tools are generally quick, 
low-cost, accessible to all students, easy to administer and 


score, and can be repeated throughout the year. Screening 
tools are designed to identify students who are not making 
expected progress and may need further assessment and/or 
instruction within the second and third tiers of the multi¬ 
tiered RTI framework to improve their skills. 

A variety of instruments are used in the RTI frame¬ 
work to identify which students need additional instruc¬ 
tion to improve their academic and behavioral skills (RTI 
Action Network, n.d.). For example, many districts utilize 
Curriculum-Based Measurement of oral reading fluency 
(CBM-R) as their screening tool. CBM-R first emerged in 
the 1970s in an effort to create measurement procedures 
that had the potential to efficiently monitor student 
progress (Deno, 1985). CBM-R requires students to read 
a passage aloud at their grade or instructional level for one 
minute. Passages are scored for the number of words read 
correctly aloud during that one minute, which results in 
an oral reading fluency number. CBM-R’s characteristics 
as an easy, quick, and inexpensive method encouraged 
calls for use of the tool for both progress monitoring and 
screening (Jenkins, Hudson, & Lee, 2007). In response to 
this growing popularity, many CBM-R products are on the 
market today including AIMSweb, DIBELS (both DIBELS 
6 th Edition and DIBELS Next), Edcheckup, Formative 
Assessment System for Teachers (FAST), and easyCBM. 

Over 30 years of research supports the reliability and 
validity of CBM-R. For example, Reschly, Busch, Betts, 
Deno, and Long (2009) conducted an extensive meta- 
analysis examining the correlational evidence between 
CBM-R and a variety of different standardized measures 
of reading achievement for students in grades one through 
six. Across all 289 correlation coefficients, the median co¬ 
efficient was 0.68 with most coefficients in the 0.60 to 0.70 
range, indicating that less than half (approximately 46%) of 
the variance in reading scores was accounted for by CBM-R 
scores (Reschly et al., 2009). Correlations with state- 
specific tests were weaker than with national tests, and the 
strength of correlations tended to decline as students in¬ 
creased in grade level. Although these overall correlations 
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were relatively strong, the pattern suggests that CBM-R 
may not be identifying a wide range of subpopulations 
of students, such as students at risk and older students. 
Technical reviews conducted by the Center on Response 
to Intervention (2014) supported these findings. While 
evidence of the reliability of CBM-R tools is compelling, 
less convincing evidence exists for validity (i.e., does this 
tool really measure reading ability?) and classification ac¬ 
curacy (i.e., are there too many false positives and/or false 
negatives?). Further, a major limitation of these studies was 
a lack of data disaggregation by demographic information 
to ensure the screening tools were accurately measuring stu¬ 
dents across different subpopulations (Reschly et al., 2009). 

These limitations notwithstanding, the What Works 
Clearinghouse released a practice guide describing five rec¬ 
ommendations for implementing RTI for student success 
(Gersten et al., 2008). The authors indicated there was 
moderate evidence to implement screening “all students 
for potential reading problems at the beginning of the 
year and again in the middle of the year [and to] regularly 
monitor the progress of students at risk for developing read¬ 
ing disabilities” (Gersten et al., 2008, p. 6). Additionally, 
there was strong evidence to “provide intensive, systematic 
instruction on up to three foundational reading skills in 
small groups to students who score below the benchmark 
score on universal screening” (Gersten et al., 2008, p. 6). 
However, these recommendations still leave many district 
representatives to question how a teacher can provide this 
intensive and systematic instruction based simply on the 
information provided through a screening tool. 

IRLA Framework and Validity Evidence 

In an ever-expanding education assessment market, 
school districts are increasingly looking for assessment prod¬ 
ucts that offer multiple functions. IRLA is one example of a 
multipurpose assessment that is a “unified standards-based 
framework for student assessment, text leveling, and curric¬ 
ulum and instruction” (American Reading Company, n.d., 
p. 1). First published in 2010, IRLA is now used in over 
4,000 schools, impacting over 900,000 students across the 
United States. 

Two unique features set IRLA apart from CBM-R and 
other assessments. First, IRLA is based on the Common 
Core State Standards (CCSS) and assesses every standard 
in literature and informational text, as well as language 
standards that are necessary for reading success for all 
grades pre-K through twelfth grade. These features are 
quite different from CBM-R, which assesses only fluency. 
Second, the IRLA is both a diagnostic and a formative 
assessment tool, allowing teachers to track progress in 
real time. Students receive points on a continuous growth 
scale in each formative-assessment conference based on the 
standards they have mastered (i.e., from a reading level of 
3.05 to 3.32 to 3.68 to 3.97 across third grade). Teachers 
assess students one-on-one to find the student’s baseline 
reading level in a 10 to 15 minute individual interview. 
Although IRLA is more expensive and time consuming 
than typical CBM-R measures (i.e., DIBELS is free and 
requires approximately three minutes per student), the 


diagnostic information ILRA provides can also guide 
instructional practices. 

Despite IRLA’s rapid growth since its inception in 
2010, the program was not included on the Center on 
Response to Intervention’s (2014) screening tools chart. 
Furthermore, little reliability and validity evidence has 
been collected to date. However, Griswold and Bunch’s 
(2014) preliminary research, commissioned by the creators 
of IRLA, examined validity evidence of the program by 
studying approximately 600 K-5 students in one Rochester, 
MN, school. Content specialists confirmed the content 
was grade-level appropriate, aligned to the CCSS, and was 
bias-free (Griswold & Bunch, 2014). Moreover, one expert 
stated that “the ILRA framework can be used to find a 
valid and reliable baseline for independent reading levels, 
PK-12” (Conradi as cited in Griswold & Bunch, 2014, 
p. 27). Survey results were also collected from teachers, 
reading specialists, and administrators regarding the use 
of the assessment tool. Overall, teachers reported IRLA 
was well-aligned to the CCSS, increased their familiarity 
with CCSS, and served a diagnostic function to help 
identify students’ learning needs (Griswold & Bunch, 
2014). Finally, concurrent validity correlation coefficients 
between the IRLA and Northwest Evaluation Association’s 
(NWEA) Measurements of Academic Progress® (MAP®) 
tool were analyzed. IRLA and MAP scores were collected 
at five time intervals from 2012-2014, and the criterion- 
related evidence correlations remained consistent: 0.88, 
0.88, 0.88, 0.88, and 0.90. The researchers also collected 
construct validity evidence by demonstrating how student 
scores increased on the IRLA between test administra¬ 
tions. None of the information was disaggregated by grade 
level, program (e.g., English Language Learners or Special 
Education), or race/ethnicity. 

This Study 

One school district turned to the research-practice 
partnership to study the concurrent validity evidence 
on the IRLA with the statewide assessment, the Oregon 
Assessment of Knowledge and Skills (OAKS), as part of 
the district’s quest for an instrument that ultimately could 
help raise its students’ test scores by using a screening tool 
matched to the student population’s unique needs. The 
public school district serves almost 11,000 ethnically and 
linguistically diverse students with nearly 75% qualifying 
for free/reduced lunch (Oregon Department of Education, 
2014). The district was facing increasing concerns about 
its low performance on the OAKS, particularly for English 
Language Learners, a rapidly increasing student subpopu¬ 
lation. Coupled with criticism surrounding the district’s 
oral reading fluency (CBM-R) screener and its validity 
and classification accuracy, the district began researching 
alternative screening instruments that could also be used 
regularly by teachers and school specialists for progress 
monitoring. This approach would provide evidence about 
whether students were moving toward meeting state 
benchmarks, as well as offering diagnostic and formative 
assessment data to guide instruction. The district selected 
IRLA as an all-in-one instrument: a screener, a diagnostic 
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assessment, and a progress monitoring tool—providing rich 
information about student reading ability and reading levels 
while using the CCSS architecture as a base. However, the 
district also wanted additional evidence documenting its 
reliability and validity. 

The district was interested in using the IRLA only if 
it were predictive of students’ performance on the OAKS. 
More specifically, the district hoped the IRLA would have 
higher correlations with the state assessment than its pre¬ 
vious screener, CBM-R. If the calculated correlation coef¬ 
ficient between the IRLA and the OAKS were moderate 
or strong, then the IRLA could be considered predictive of 
OAKS, especially its performance by grade level, program 
(e.g., ELL, Special Education), and race/ethnicity to better 
examine for whom the concurrent validity coefficients were 
highest. These data would add evidence to the validity of 
the IRLA, as both tools measure reading comprehension. 

Methods 

Our data analyses examined the relationship between 
the reading scores of two different concurrently adminis¬ 
tered reading assessments, the IRLA and the OAKS, with 
students in grades three through five in one school district. 
Both the IRLA reading level score and the OAKS standard 


score are standardized, interval-level variables; therefore, 
Pearson product-moment correlation coefficients were cal¬ 
culated to examine the relationship between the IRLA and 
OAKS. Percent exact agreement was also used to measure 
the categorization accuracy of students being categorized as 
either meeting benchmark or not meeting benchmark by 
the two assessments. This was calculated by dividing the 
number of matches by the total number of opportunities 
to match. The data were further disaggregated to examine 
the relationships among the different demographic groups. 
The practical significance of the relationships was exam¬ 
ined through It effect sizes. 

Participants 

Participants included 2,303 students attending 11 
elementary schools in one school district in the Pacific 
Northwest: 803 third-grade students (35%), 720 fourth- 
grade students (31%), and 780 fifth-grade students (34%). 
Thirty-seven students were excluded from the sample 
because they did not complete the OAKS, and instead 
they completed the alternative state assessment (36 of the 
37 students were receiving special education services). 
Table 1 provides additional participant demographics, 
including gender and race/ethnicity information. 


Table 1 

Participant Demographics 


Demographic Variable 

Third Grade 

Fourth Grade 

Fifth Grade 

Overall 

Gender 





Male 

52% 

50% 

53% 

52% 

Female 

48% 

50% 

47% 

48% 

Program 





Receiving Talented and Gifted (TAG) 
Services 

9% 

6% 

9% 

8% 

Receiving Special Education Services 

13% 

12% 

16% 

14% 

Being Monitored for ELL Services (exited) 

13% 

13% 

20% 

15% 

Receiving ELL Services 

39% 

31% 

26% 

32% 

Race/Ethnicity 





American Indian/Alaskan Native 

<1% 

<1% 

<1% 

<1% 

Asian 

6% 

8% 

7% 

7% 

Black/African American 

9% 

9% 

8% 

9% 

Latino/Hispanic 

45% 

42% 

45% 

44% 

Multi-Racial 

6% 

5% 

6% 

6% 

Native Hawaiian/Pacific Islander 

3% 

3% 

3% 

3% 

White 

31% 

34% 

31% 

32% 
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Instruments 

IRLA. IRLA provides an interval-level score on a 
growth scale continuum. The IRLA mean scores and IRLA 
percent at each performance level are displayed in Table 2. 
Students are also assigned a risk category based on their 
score. Students who are “on grade level” or “benchmark” 
are “low risk” and read within their grade level (i.e., a 
third-grade student has a score of 3.00 and above), while a 
student with “some risk” reads up to one year below grade 
level (i.e., a third-grade student has a score of 2.00-2.99), 
and a student “at risk” reads one or more years below grade 
level (i.e., a third-grade student has a score of 1.99 or less). 
The percent of students in each category in each grade is 
also reported in Table 2. 

OAKS. The Oregon Assessment of Knowledge and 
Skills (OAKS; Oregon Department of Education, 2010) is 
the standardized test for students in grades three through 
five aligned to the 2002 State English Language Arts 
content standards. A lengthy technical manual details 
reliability and validity evidence, including high concur¬ 
rent validity scores with the California Achievement Test 
(r = 0.75 - 0.80), the Iowa Test of Basic Skills® (r = 0.78 - 
0.84), the NWEA Subject Tests (r = 0.73 - 0.81), and the 
Lexile Scale® for reading (r = 0.76 - 0.77). Although the 
CCSS were adopted in the state in October of 2010, full 

Table 2 


implementation of CCSS occurred in the 2014-15 school 
year, at which point Smarter Balanced was to be used as 
the statewide assessment. During this 2013-14 school year, 
the district was in the process of converting to CCSS while 
still assessing the state content standards via the OAKS. An 
analysis of a state-conducted crosswalk of state standards 
and CCSS showed that, for the most part, the two sets of 
standards were fairly well aligned (Oregon Department of 
Education, 2013a, 2013b). The standards were partially 
or strongly aligned for 72% of the third-grade English 
Language Arts Standards and for 82% of the third-grade 
Mathematics Standards, for example. Smarter Balanced, 
which is aligned to CCSS, replaced OAKS for the 2014- 
2015 school year. The OAKS cut score for meeting bench¬ 
mark increased five points each year: from 211 in third 
grade to 216 in fourth grade to 221 in fifth grade. The 
percent of students in each category in each grade is also 
reported in Table 3. 

Results 

The overall correlation between OAKS and IRLA 
data for all students was 0.766 (p < .001), indicating that 
approximately 59% of the variance in OAKS scores is ac¬ 
counted for by the IRLA scores. Overall, IRLA appeared 
to be a strong predictor of OAKS. Across all students, 80% 


IRLA Performance Results 


Grade 

n 

IRLA 

Mean Score 

IRLA Percent 

At Risk 

IRLA Percent 
Some Risk 

IRLA Percent 
Low Risk 

Third Grade 

803 

2.78 (SD = 1.13) 

18% 

32% 

50% 

Fourth Grade 

720 

3.48 (SD = 1.24) 

34% 

23% 

43% 

Fifth Grade 

780 

3.98 (SD = 1.59) 

45% 

22% 

32% 

Overall 

2,303 

3.41 (SD = 1.42) 

32% 

26% 

42% 


Table 3 

OAKS Performance Results 

Grade 

n 

OAKS 

Mean Score 

OAKS Not Meeting 
Benchmark 

OAKS Meeting 
Benchmark 

Third Grade 

803 

208.34 (SD = 11.73) 

56% 

44% 

Fourth Grade 

720 

215.27 (SD = 10.68) 

50% 

50% 

Fifth Grade 

780 

219.15 (SD = 10.28) 

54% 

46% 

Overall 

2,303 

214.17 (SD = 11.83) 

53% 

47% 
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of students were categorized as either meeting benchmark 
or not similarly on IRLA and on OAKS (see Table 4). 

Concurrent Validity by Grade 

Because one of the variables (IRLA) uses a continuous 
score across the grades, while the other variable (OAKS) 
uses a benchmark score that grows slightly but is fairly 
consistent across the three grades, it is important to inves¬ 
tigate the correlation coefficients by grade. Table 4 shows 
the Pearson _product-moment correlation coefficients, the 
associated R effect sizes, and the classification accuracy of 
students’ meeting/not meeting benchmarks as measured 
by percent exact agreement for each of the three grades. 
The effect sizes of the correlations are all large as defined 
by Cohen (1988). It appears that the correlation is highest 
for fourth-grade students, yet the prediction matching of 
benchmarks is the highest for third-grade students. 

Concurrent Validity by Program 

Table 5 shows the concurrent validity data by pro¬ 
gram. The effect sizes of the correlations are all large, 
except for ELL students who have exited and are being 
monitored, which is a moderate effect size (Cohen, 


1988). As mentioned previously, the values for students 
receiving special education services may be overinflated 
because of the removal of 10% of the special education 
students. While the percent exact agreement for Talent¬ 
ed and Gifted (TAG) students is high, the correlation 
is low. Conversely, it appears that both the correlation 
coefficients and the percent exact agreement are lower for 
ELL students, both those actively receiving services and 
those being monitored; however, these scores are lowest 
for monitored ELL students. 

Concurrent Validity by Race/Ethnicity 

Table 6 shows the concurrent validity data by race/ 
ethnicity. The effect sizes of the correlations are all large 
(Cohen, 1988). American Indian/Alaskan Native students 
were not included in this analysis because the total number 
of students was fewer than 10, and the correlation for Native 
Elawaiian/Pacific Islander students must be interpreted 
cautiously as well due to the small sample size. Multiracial 
students had the highest correlation coefficient, while White 
students had the largest percentage exact agreement. Both 
the correlation coefficient and the percent exact agreement 
were smallest for Asian students. 


Table 4 

Concurrent Validity of IRLA and OAKS by Grade 


Grade 

n 

r 

R 2 

Percent Exact 
Agreement 

Third Grade 

803 

0.713* 

51% 

83% 

Fourth Grade 

720 

0.775* 

60% 

79% 

Fifth Grade 

780 

0.751* 

56% 

77% 

Overall 

2,303 

0.766* 

59% 

80% 

*p < .001 





Table 5 





Concurrent Validity o 

f IRLA and OAKS by Program 




Program 

n 

r 

R 2 

Percent Exact 
Agreement 

TAG 

178 

0.534* 

29% 

91% 

Special Education 

324 

0.789* 

62% 

90% 

ELL Monitored 
(exited) 

350 

0.477* 

23% 

71% 

ELL Active 

736 

0.644* 

41% 

86% 


*p < .001 
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Table 6 


Concurrent Validity of IRLA and OAKS by Race/Ethnicity 


Race/Ethnicity 

n 

r 

R2 

Percent Exact 
Agreement 

Asian 

157 

0.684* 

47% 

73% 

Black/African American 

197 

0.771* 

59% 

77% 

Latino/Hispanic 

1,011 

0.751* 

56% 

81% 

Multi-Racial 

127 

0.778* 

61% 

74% 

Native Hawaiian/Pacific Islander 

62 

0.697* 

49% 

87% 

White 

741 

0.753* 

57% 

80% 


*p < .001 


Discussion 

This study examines the concurrent validity of the 
IRLA reading assessment with the OAKS state standard¬ 
ized reading test, disaggregating the data by grade level, 
program, and race/ethnicity to better examine for whom 
the concurrent validity coefficients are highest. We add 
to the limited research base of evidence that the IRLA 
tool may be an important instrument for bridging the 
gap between screening and providing intensive, systematic 
instruction as detailed by the What Works Clearinghouse 
(Gersten et ah, 2008), especially for ethnically diverse and 
socioeconomically disadvantaged student subpopulations. 

Our results parallel those conducted by Measurement 
Incorporated (Griswold & Bunch, 2014), although we found 
slightly lower correlations between IRLA and standardized 
reading tests. Our study was conducted with a greater num¬ 
ber of schools (i.e., 11 schools vs. 1 school) and likely across 
a more diverse population. One potential reason the correla¬ 
tions between MAP and IRLA (Griswold & Bunch, 2014) 
may have been higher than the correlations described in 
this study is that the OAKS assessment is summative where 
both MAP and IRLA assessments are designed to measure 
growth. The OAKS also has a small range of possible scores, 
which may cause a range limitation. Additionally, the results 
could be underestimated due to the misalignment of stan¬ 
dards between OAKS and IRLA. Future studies between 
IRLA and Smarter Balanced and/or other CCSS-aligned 
measures may produce higher coefficients. 

Additionally, the correlation coefficients reported here 
are higher than the median coefficient of 0.68 reported 
in the meta-analysis conducted by Reschly and colleagues 
(2009) on CBM-R. While approximately 49% of the vari¬ 
ance in reading scores was accounted for by CBM-R, 59% 
of the variance in state standardized reading score was 
accounted for by IRLA. Furthermore, this study is the first 
attempt to disaggregate the validity evidence across grade, 
program, and race/ethnicity to better understand how the 


assessment functions across varying demographic catego¬ 
ries. This strategy is severely lacking in the literature on 
CBM-R as well as with IRLA. It is imperative that accurate 
assessment tools, validated for all grades, programs, races/ 
ethnicities, and other student subgroups, be utilized to best 
reach students of various demographics. While RTI creates a 
framework for closing the achievement gap, tools like IRLA 
provide methods of making decisions based on individual 
student data on what types of high-quality, evidence-based 
instruction is necessary. 

Limitations and Future Research 

There are several limitations with this dataset. First, 
as discussed previously, 10% of students receiving special 
education services (i.e., 36 of the 360 total students) were 
removed from this analysis because they completed an alter¬ 
native state assessment instead of the OAKS. Therefore, all 
information regarding students receiving special education 
services should be interpreted cautiously. Second, these data 
are from only one district in one area of Oregon. Future 
research should investigate the validity evidence of IRLA in 
wider, more diverse populations. Third, this school district 
was in its first year of implementation of IRLA. IRLA is 
quite different from CBM-R, which the district was using 
previously. Program fidelity is a concern as teachers learned 
to use a new assessment tool that required some adaptation 
and a learning curve. Future research should investigate 
not only teacher implementation over time and how imple¬ 
mentation affects use of the tool and student growth, but 
also the relationship between the new Smarter Balanced 
assessment, CBM-R, and IRLA. 

Implications for Practice 

Overall, the IRLA appeared to predict state standard¬ 
ized reading scores well, and this prediction appeared to 
remain consistent when disaggregating across subgroups. 
These results have initial implications for practice, both 
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for the participating district and other districts. Although 
assessments that use curriculum-based measures have many 
benefits, including lower cost, efficiency, etc., there are other 
assessment options available for districts, particularly for 
those districts with an ethnically diverse and socioeconomi¬ 
cally disadvantaged student population. In this one particu¬ 
lar district, interviews with district personnel revealed that 
they were exceptionally pleased with the first year’s results 
of the IRLA. For example, one district administrator said, 
“Our teachers are becoming expert teachers of reading- 
many stating that they have never so deeply understood 
their students’ abilities and needs.” 

Further, preliminary indicators from the second year 
of implementation indicate higher IRLA scores than the 
previous year at the same time in the school year. Thus, 
district personnel are hopeful that these results will also 
be reflected on the state standardized tests. More research 
is necessary to ensure this pattern holds. Finally, district 
leadership believes IRLA provides teachers and principals 
with formative assessment data that can be immediately 
used and tracked to make instructional and leadership 
decisions, unlike CBM-R could do previously. Ongoing 
data collection will be interesting to study, both as teachers 
gain familiarity with using the tool (i.e., their use of interim 
scores increases) and as the district moves from measuring 
student progress with OAKS to using the Smarter Balanced 
assessment. 
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